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Abstract 

For large scale symmetric discrete ill-posed problems, MINRES and MR-11 are often used 
iterative regularization solvers. We call a regularized solution best possible if it is at least as 
accurate as the best regularized solution obtained by the truncated singular value decomposition 
(TSVD) method. In this paper, we analyze their regularizing effects and establish the following 
results: (i) the filtered SVD expression are derived for the regularized solutions by MINRES; 
(ii) a hybrid MINRES that uses explicit regularization within projected problems is needed to 
compute a best possible regularized solution to a given ill-posed problem; (iii) the kth iterate 
by MINRES is more accurate than the (fc — l)th iterate by MR-II until the semi-convergence 
of MINRES, but MR-II has globally better regularizing effects than MINRES; (iv) bounds are 
obtained for the 2-norm distance between an underlying fc-dimensional Krylov subspace and 
the fc-dimensional dominant eigenspace. They show that MR-II has better regularizing effects 
for severely and moderately ill-posed problems than for mildly ill-posed problems, and a hybrid 
MR-II is needed to get a best possible regularized solution for mildly ill-posed problems; (v) 
bounds are derived for the entries generated by the symmetric Lanczos process that MR-II is 
based on, showing how fast they decay. Numerical experiments confirm our assertions. Stronger 
than our theory, the regularizing effects of MR-II are experimentally shown to be good enough 
to obtain best possible regularized solutions for severely and moderately ill-posed problems. 
Keywords: Symmetric ill-posed problem, regularization, partial regularization, full regulariza¬ 
tion, semi-convergence, MR-II, MINRES, LSQR, hybrid. 

Mathematics Subject Classifications (2010): 65F22, 65J20, 15A18. 


1 Introduction 

Consider the large scale discrete linear ill-posed problem 

Ax = b, (1.1) 

where A is symmetric and extremely ill conditioned with its singular values decaying gradually to 
zero without a noticeable gap. This kind of problem arises in many science and engineering areas 
[10]. The right-hand side b is noisy and typically affected by a white noise, caused by measurement, 
modeling or discretization errors, i.e., 

b = b + e, 
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where e € M” represents a white noise vector and h G M” denotes the noise-free right-hand side, and 
it is supposed that ||e|| < ||6||. Because of the presence of noise e in 6 and the high ill-conditioning 
of A, the naive solution Xnaive = A~^h of (1.1) is far from the true solution xtme = A~^h and 
meaningless. Therefore, one needs to use regularization to determine a regularized solution so that 
it is close to xtme = A~^b as much as possible [8, 10]. 

For A symmetric, its SVD is closely related to its spectral decomposition as follows: 

A = VAV'^ = (1.2) 

where [/ = (ui,U 2 , ■ ■ ■, Un) = VVt. and V = (vi,V 2 , ■ ■ ■, Vn) are orthogonal, whose columns are the 
left and right singular vectors of A, respectively, the diagonal matrix E = diag(cJi, cj 2, ..., cr„) with 
the singular values labeled as cJi > £72 > • • • > > 0, = diag(±l) is a signature matrix such 

that cjj = |Aj| with the Aj the eigenvalues of A, and A = diag(Ai, A2 ,..., Xn)- Obviously, Ui = 
with the ± sign depending on fl. With (1.2), we can express the naive solution of (1.1) as 

n n 7-’? n q-' n q-’ 

E v- b 0 , e , ^ /-, oa 

j ~r Vi -|- y ^ —- Vi — Xtrue -|- ^ ^ —- Vi- (1’3) 

1=1 1=1 1=1 1=1 

Throughout the paper, we assume that b satisfies the discrete Picard condition [8, 10]: On 
average, the coefficients \ufb\ = Ivfbl decay faster than the singular values ai = jAjj. This is a 
necessary hypothesis that controls the size of ||xirue|| and makes regularization possible to find 
meaningful approximations to xtme [8, 10]. For the sake of simplicity, we assume that they satisfy 
a widely used model [8, 10]: 

\ vfb \= \Xi\^^^, /? > 0, 7 = 1,2,... ,n. (1.4) 

Similar to the truncated SVD (TSVD) method [8, 10], for A symmetric, a truncated spectral 
decomposition method obtains the TSVD regularized solutions 

^ Tb ^ Tb 

= V' ^ —Ui = A\b, A: = 1, 2,... , n, (1.5) 

*=i *=i 

where Aj- = Uk^kVk with Uk and I4 the first k columns of U and V, respectively, = 
diag(cJi,..., fjfc), and f the Moore-Penrose generalized inverse of a matrix. Obviously, Xk is the 
minimum 2-norm least squares solution of the perturbed problem that replaces A in (1.1) by its 
best rank k approximation Ak- 

Let kQ denote the transition point such that j v'^^b |>| \ and j j, which 

divides the eigenvalues Aj or equivalently the singular values Uj = jAj] into the dominant or large 
ones for i < ko and the small ones for i > k^. It is known from [8, p. 176] and [10, p. 86-88] that the 
best TSVD regularized solution is Xk^-, which consists of the k^ dominant SVD components of A, 
i.e., the dominant spectral components corresponding to the first fcg large eigenvalues in magnitude 
when A is symmetric. A number of approaches have been proposed for determining /cQ) such as 
discrepancy principle, the discrete L-curve criterion and the generalized cross validation (GCV); see, 
e.g., [1, 4, 8, 15, 21] for comparisons of the classical and new ones. In our numerical experiments, 
we do this using the L-curve criterion in the TSVD method and Krylov iterative methods. 

The TSVD method is important in its own right and plays a central role in analyzing the 
standard-form Tikhonov regularization [8, 10]. It and the standard-form Tikhonov regularization 
expand their solutions in the basis vectors Vi and produce very similar solutions with essentially 
the minimum 2-norm error; see [8, p. 109-111] and [10, Sections 4.2 and 4.4]. Therefore, the TSVD 
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method can get a best regularized solution to (1.1), and it has long been used as a general-purpose 
reliable and efficient numerical method for solving a small and/or moderate sized (1-1) [8, 10]. As a 
result, we will take the TSVD solution XkQ as a standard reference when assessing the regularizing 
effects of iterative solvers and accuracy of iterates under consideration in this paper. 

For (1.1) is large, it is generally impractical to compute the spectral decomposition of A. In 
this case, one typically solves it iteratively via some Krylov subspace methods [8, 10]. For (1.1) 
with a general matrix A, the mathematically equivalent LSQR [19] and CGLS [2] have been most 
commonly used for years and have been shown to have intrinsic regularizing effects [6, 8, 10]. They 
exhibit the semi-convergence; see [6], [8, p. 135], [10, p. 110]: The iterates tend to be better and 
better approximations to the exact solution xtme and their norms increase slowly and the residual 
norms decrease. In later stages, however, the noise e starts to deteriorate the iterates, so that 
they will start to diverge from xtme and instead converge to Xnaive, while their norms increase 
considerably and the residual norms stabilize. For LSQR, the semi-convergence is due to the fact 
that the projected problem at some iteration starts to inherit the ill-conditioning of (1.1), that is, 
the noise progressively enters the solution subspace, so that a small singular value of the projected 
problem appears and the regularized solution is deteriorated [8, 10]. 

As far as an iterative solver for solving (1.1) is concerned, a central problem is whether or not 
it has already obtained a best possible regularized solution at semi-convergence. Here, as defined 
in the abstract, a best possible regularized solution means that it is at least as accurate as the best 
TSVD solultion x^q- This problem has been intensively studied but has had no definitive solutions. 
For Krylov iterative solvers, their regularizing effects critically rely on how well the underlying 
/c-dimensional Krylov subspace captures the k dominant right singular vectors of A [8, 10]. The 
richer information the Krylov subspace contains on the k dominant right singular vectors, the less 
possible it is that the resulting projected problem has a small singular value. That is, the solvers 
capture the large SVD components of A more effectively, and thus have better regularizing effects. 

To precisely measure the regularizing effects, we introduce the term of full or partial regu¬ 
larization. If a pure iterative solver itself computes a best possible regularized solution at semi¬ 
convergence, it is said to have the full regularization; in this case, no additional regularization is 
necessary. Otherwise, it is said to have the partial regularization; in this case, a sophisticated 
hybrid variant is needed that combines the solver with some additional regularization in order to 
improve the accuracy of the regularized solution by the iterative solver at semi-convergence [8, 10]. 
It appears that the regularizing effects are closely related to the degree of ill-posedness of the 
problem. To this end, we introduce the following definition of the degree of ill-posedness, which 
follows Hofmann’s book [12] and has been commonly used in the literature, e.g., [8, 10]: If there 
exists a positive real number a such that the singular values satisfy aj = 0(j““), the problem is 
termed as mildly or moderately ill-posed if a < 1 or a > 1; if cjj = with a > 0 consider¬ 

ably, j = 1, 2,... , n, the problem is termed severely ill-posed. Clearly, the singular values aj of a 
severely ill-posed problem decay exponentially at the same rate , while those of a moderately or 
mildly ill-posed problem decay more and more slowly at the decreasing rate approaching 

one with increasing j, which, for the same j, is smaller for the moderately ill-posed problem than 
it for the mildly ill-posed problem. 

For A symmetric, its /c-dimensional dominant eigenspace is identical to the /c-dimensional dom¬ 
inant left and right singular subspaces. In this case, MINRES and its variant MR-H are natural 
alternatives to LSQR and CGLS [5, 7, 10]. MR-H was originally designed for solving singular and 
inconsistent linear systems, and it uses the starting vector Ab and restricts the resulting Krylov 
subspace to the range of A. Thus, the iterates are orthogonal to the null space of A, and MR-H 
computes the minimum 2-norm least squares solution [3, 5]. For (1.1), we are not interested in such 
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solution but a regularized solution that is close to xtrue as much as possible. MINRES and MR-II 
have been shown to have regularizing effects and exhibit semi-convergence [11, 14, 16], and MR-II 
usually provides better regularized solutions than MINRES. Intuitively, this is because the noise e 
in the initial Krylov vector Ab is filtered by multiplication with A [6, 14]. Different implementations 
associated with MR-II have been studied [5, 17]. 

In this paper, we first prove that the MINRES iterates are filtered SVD solutions, similar to 
the form of the LSQR iterates. Based on this result, we show why MINRES, in general, has 
only the partial regularization, independent of the degree of ill-posedness of (1.1). As a result, 
a hybrid MINRES that combines MINRES with a regularization method applied to the lower 
dimensional projected problems should be used to compute a best possible regularized solution; see 
[10, Section 6.4] for details. Afterwards, we take a closer look at the regularization of MINRES 
and MR-II in more detail, which, from a new perspective, shows that MINRES has only the 
partial regularization. We prove that, though MR-II has globally better regularizing effects than 
MINRES, the /cth MINRES iterate is always more accurate than the {k — l)th MR-II iterate 
until the semi-convergence of MINRES. In a manner different from those used in [14, 16], we 
then analyze the regularizing effects of MR-II and draw some definitive conclusions. We establish 
bounds for the 2-norm distance between the underlying fc-dimensional Krylov subspace and the k- 
dimensional dominant eigenspace. The bounds indicate that the Krylov subspace better captures 
the fe-dimensional dominant eigenspace for severely and moderately ill-posed problems than for 
mildly ill-posed problems. As a consequence, MR-II has better regularizing effects for the first two 
kinds of problems than for the third kind, for which MR-II has only the partial regularization. We 
then use the results to derive an estimate for the accuracy of the rank k approximation generated 
by MR-II to A. Finally, we derive estimates for the entries generated by the symmetric Lanczos 
process that MR-II is based on, and show how fast they decay. 

The paper is organized as follows. In Section 2, we describe MINRES and MR-II. In Section 
3, we prove that the MINRES iterates are filtered SVD solutions, followed by an analysis on the 
regularizing effects of MINRES. In Section 4, we compare the regularizing effects of MINRES and 
MR-II, and shed light on some new features of them. In Section 5, we present our theoretical results 
on MR-II with a detailed analysis. In Section 6, we numerically confirm our theory that MINRES 
has only the partial regularization for a general ill-posed problem and its hybrid variant is needed. 
Also, we experimentally illustrate that MR-II has the full regularization for severely and moderately 
ill-posed problems, which is stronger than our theory, and it has the partial regularization for mildly 
ill-posed problems. We also compare MR-II with LSQR, demonstrating that MR-II is as effective 
as and at least twice as efficient as LSQR. We conclude the paper in Section 7. 

Throughout the paper, we denote by ]Ck{C,w) = span{w,Cw, ... the A:-dimensional 

Krylov subspace generated by the matrix C and the vector w, by || • || and || • \\f the 2-norm of a 
matrix or vector and the Probenius norm of a matrix, respectively, and by I the identity matrix 
with order clear from the context. 

2 MINRES and MR-II 

MINRES [20] is based on the symmetric Lanczos process that constructs an orthonormal basis 
of the Krylov subspace /Cfc(A, 6). Let qi = 6/11^11- The fc-step symmetric Lanczos process can be 
written in the matrix form 

-^Qk Qk+l^ki 

where Qk+i = (^ij 92 ) ■ ■ ■ j 9fc-i-i) has orthonormal columns which form ICk{A, b), and Tk € 
is a tridiagonal matrix with its leading k x k submatrix symmetric. 
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At iteration k, MINRES solves ||6 —|| = ||6 —Ax|| for the iterate 

with 

y{k) ^ argmin ||||6||ei -fky\\, (2.1) 

ySK*" 

where ei is the first canonical vector of dimension k + 1. For our analysis purpose, it is important 
to write 

x^^'> = QkflQl+^b, ( 2 . 2 ) 

which is the minimum 2-norm least squares solution of the perturbed problem that replace A in 
(1.1) by its rank k approximation Qk+iTkQk ■ 

MR-II [5] is a variant of MINRES applied to JCk{A, Ab) which excludes the noisy b. The method 
is based on the /c-step symmetric Lanczos process 

AQk = Qk+iTk, (2.3) 

where Qk+i = (^ij (i'2) ■ ■ ■ j Q'fc+i) has orthonormal columns with qi = A5/||A6||, Tk G 
is a tridiagonal matrix with the diagonals ctj, the subdiagonals /3j > 0, i = 1,2, ...,A: and the 
superdiagonals /3j > 0, i = 1,2,..., A: — 1, and the hrst k rows of Tk is symmetric. The columns of 
Qk form an orthonormal basis of JCk{A, Ab). Mathematically, since the eigenvalues of A are simple 
and Ab has nonzero components in the directions of all the eigenvectors Vi of A, the Lanczos process 
can be run to n steps without breakdown, i.e., /3fe > 0, k = 1,2,... ,n — 1 and (3n = 0. 

At iteration k, MR-II solves ||6 —for the iterate = QkU^^'^ 
with 

yW = arg min ||6 - Qfc+iTfcy||. (2.4) 

j/GK'= 

Similar to (2.2), we have the expression 


= QkTlQl+^b, (2.5) 

which is the minimum 2-norm least squares solution of the perturbed problem that replace A in 
(1.1) by its rank k approximation Qk+i^kQk ■ 

The significance of (2.2) and (2.5) lies that the MINRES and MR-II iterates are the minimum 
2-norm least squares solutions of the perturbed problems that replace A in (1.1) by its rank k 
approximations Qk+iTkQk Qk+iTkQk ^ respectively, whose k nonzero singular values are just 
those of Tk and Tk, respectively. If the singular values of Tk or Tk approximate the k large singular 
values of A in natural order for fc = 1, 2,..., fco, then Qk+iTkQ]^ and Qk+iTkQ]^ are near best rank 
k approximations to A with accuracy similar to that of the best rank k approximation A^. If this 
is the case, MINRES and MR-II must have the full regularization, and x^^°^ and are best 
possible regularized solutions and are as accurate as the best TSVD regularized solution XkQ- 


3 The regularizing effects of MINRES 


Similar to the CGLS and LSQR iterates [8, p. 146], we can establish the following result on the 
MINRES iterates. 


Theorem 3.1. For MINRES to solve (1.1) with the starting vector qi = fe/||6||, the kth iterate x^^'^ 
has the form 




E/i 


(fc) 


vfb 


Vi, 


i=l 


(3.1) 
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/, X k ^ . 

where the filters /^ = 1 — 0 ^ (k) \ i = 1,2, ...,n with |Ai| > IA 2 I > ■■■ > |An| > 0, and 

i=i 

j = 1)2, are the harmonic Ritz values of A with respeet to }Ck{A,b) and labeled as 

\Gi^\ > 1^2 >■■■> > 0 . 

Proof. From [18], the residual = b — Ax^^'l of the MINRES iterate can be written as 

fW = Xk{A)b, (3.2) 


where the residual polynomial Xk{t) has the form 

k nW _ f 

=n 

i=i 

with the the harmonic Ritz values of A with respect to ICk{A, b). Prom (3.2), we get 

= {I - Xk{A))A-^b. 

Substituting A = VAV^ into the above gives 


^(k) 


E/f’ 



where 


/f’=i-n- 

j=i 


g(fc) 


-A,: 


Xk) 


i = 1,2,... ,n. □ 


Relation (3.1) shows that the MINRES iterate has a hltered SVD expansion. Eor a general 
symmetric A, the harmonic Ritz values have an attractive feature: they usually favor extreme 
eigenvalues of A, provided that a Krylov subspace contains substantial information on all the 
eigenvectors Vi [18]. In our current context, if at least a small harmonic Ritz value in magnitude 
starts to appear for some k < ko, i.e., < |Afco+i|, the corresponding filter factors f-^\ i = 

k + 1,... ,n, are not small, meaning that is already deteriorated. On the other hand, if no small 
harmonic Ritz value in magnitude appears before k < ko, the x^^^ are expected to become better 
approximations to xtme until k = k^. Unfortunately, since ICk{A,b) includes the noisy b = b + e, 
which contains non-negligible components of Vi corresponding to small eigenvalues Aj, it is generally 
possible that a small harmonic Ritz value can appear for k < k^. This demonstrates that, in general, 
MINRES only has the partial regularization and cannot obtain a best possible regularized solution. 


4 Regularization relationships between MINRES and MR-II 

It was known a long time ago that MR-II has better regularizing effects than MINRES, that is, 
MR-II obtains a more accurate regularized solution than MINRES does [5]. Such phenomenon is 
simply due to the fact that ICk{A,b) for MINRES includes the noisy b and ICk{A,Ab) for MR-II 
contains less information on Vi corresponding to small eigenvalues in magnitude since the noise e 
in the starting vector Ab is filtered by multiplication with A. Previously, we have given an analysis 
on the regularizing effects of MINRES and shown that a hybrid MINRES is generally needed for 
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an ill-posed problem, independent of the degree of ill-posedness of (1.1). Next we shed more light 
on the regularization of MINRES, compare it with MR-II, and reveal some new features of them. 

To simplify our discussions, without loss of generality, we can well assume that for a standard 
nonsingular linear system, the smaller residual, the more accurate the approximate solution is. 
Given the residual minimization property of MINRES and MR-II, one might be confused that, 
since lCk-i{A, Ab) C JCk{A,b), the kth. MINRES iterate should be at least as accurate as the 
{k — l)th MR-II iterate This is true for solving the standard linear system where the right- 

hand side is supposed to be exact, but it is nontrivial and depends for solving an ill-posed problem, 
for which the b is noisy and we are concerned with regularized approximations to the true solution 
xtrue other than the naive solution Xnaive- Our previous analysis has shown that a small harmonic 
Ritz value \0^'^\ < crkQ+i = |Afco+i| generally appears for MINRES before some iteration k < ko, 
causing that MINRES has only the partial regularization. On the other hand, however, note that 
the regularized solutions x^^^ by MINRES converge to xtme until the semi-convergence of MINRES. 
As a result, because JCk-i{A, Ab) C /Ck(A,b), the kth MINRES iterate x^^^ is more accurate than 
the {k — I)th MR-II iterate only until the semi-convergence of MINRES. 

We can also explain the partial regularization of MINRES in terms of the rank k approximation 
Qk+iTkQk to A as follows: Since the /c-dimensional dominant eigenspace of A is identical to its 
fc-dimensional dominant left and right singular subspaces, ICk{A, b) contains substantial information 
on all the Vi. As a result, it is generally possible that the projected matrix T), has a singular value 
smaller than (Tko+i for some k < k^. This means that Qk+iThQ^ is a poor rank k approximation to 
A, causing, from (2.1), that ||xi^i|| = = ll&ll II is generally large, i.e., is already 

deteriorated. Conversely, if no singular value of Tk is smaller than (Tk^+i and the semi-convergence 
of MINRES does not yet occur, the MINRES iterate x^^i should be at least as accurate as the 
MR-II iterate because of lCk-i{A,Ab) C K-k{A,b). 

In summary, we need to use a hybrid MINRES with the TSVD method or the standard-form 
Tikhonov regularization applied to the projected problem in (2.1) to expand the Krylov subspace 
until it contains all the k^ dominant spectral components and a best regularized solution is found, 
in which the additional regularization aims to remove the effects of small singular values of Tfc+i, 
similar to the hybrid LSQR see [10, Section 6.4], 

5 Regularizing effects of MR-II 

Before proceeding, we point out that, unlike (3.1) for the MINRES iterates x^^\ we have found 
that the MR-II iterates x^^'> do not have filtered SVD expansions of similar form. Even so, we 
can establish a number of other results that help to better understand the regularization of MR- 
II. We first investigate a fundamental problem: how well does the underlying subspace JCk{A,Ab) 
capture the k dimensional dominant eigenspace of A? This problem is of basic importance because 
it critically affects the accuracy of Qk+iTkQ^ as a rank k approximation to A. 

In terms of the definition of canonical angles 0(A’,T) between the two subspaces X and y of 
the same dimension [22, p. 250], we present the following result. 

Theorem 5.1. Let A = = Vbe defined as (1.2), and assume that the singular values 

of A are aj = |Aj| = 0{e~^^) with a > 0. Let Vk = span{Vk} be the k-dimensional dominant 
spectral subspace spanned by the columns ofVk = {vi,V 2 , ■ ■ ■ ,Vk), and = }Ck{A,Ab). Then 

l|dne(Vt,v|)|| = ^J%L (5.1) 

VI + ||Afc||"= 
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with the {n — k) X k matrix to be defined by (5.3) and 

\vjb\ 


lA 


k\\F < 


|Afc+i 


max 


■j=k+i 


y/k{n - k) (l + C»(e““)) , /c = 1, 2,... , n - 1. (5.2) 


|Afc| |?;J6| 

Proof. Note that }Ck{A., AV'^b) is spanned by the columns of the nxk matrix DB^ with 


D = diag [Xivf b) , = 


Partition D and Bk as follows: 


D = 


Di 0 

0 D 2 


/ 1 

Ai 



1 

A 2 


\ A:—1 
^2 

V 1 

An 



Bk 


Bki 

Bk2 



where Since B^i is a Vandermonde matrix with Aj distinct for 1 < j < A:, it is 

nonsingular. Noting }Ck{A,Ab) = V/Cfc(A, AV^6), we have 


lCk{A, Ab) = span{VDBk} = span < V 


with 


DiBki 

D2Bk2 


Afc = D2Bk2Bj^^D^ ^ 


= span< V 


I 

Afc 


(5.3) 


Dehne Zu = V 


I 

Ai. 


. Then Z'^Zj. = / + A^A^, and the columns of Zk{Z^Zk) 2 form 


an 


orthonormal basis of Vt 


Write V = {Vk,V^). By dehnition, we obtain 

||sin0(Vfe,VDII= {V^fZk{ZlZk)-"^ 


Afc )^I + ^kAk)- 

= ||Afc(/ + A^Afc)-V2|| 

_ ||Afc|| 

\/l + l|Afc|p 

which proves (5.1). 

We next estimate ||Afc|| and establish upper bound for the right-hand side of (5.1). We have 


lAfcll < ||A||^ = \\D2Bk2Bfi^Df^\\p < IIZI 2 II \\Bk2Bfi^\\p \\DI 

\vjb\ 


-i| 


max 


■j=k+i I 


|Afc+i 
|Afc| min^^;^ 


iBuoB 


-i| 


fes-Ofci IIf ■ 


(5.4) 


We now estimate ||j 7 - It is easily justified that the ith column of consists of the 

coefficients of the Lagrange polynomial 


















that interpolates the elements of the ith canonical basis vector at the abscissas Ai,..., A^. 


Consequently, the zth column of is 

Bk2B^le^^ = (Lf )(Afc+i),..., L[^\Xn)Y, 


from which we obtain 


Bk2Bu^ — 


( Lf\\k+i) L^2\^k+i) ... Lf\\k+i)\ 
Lf\\k+2) 4"^(Afc+2) ... 4"^(Afc+2) 


V Lf\K) Lf\Xn) 


Lt\Xn) ) 


(5.5) 


For a fixed A satisfying |A| < |Afc+i|, let io = arg max |Lj-^^(A)|. Then we have 

i=l,2,...,k 


Xk)/\\\ _ 


k 

n 

A-A, 

k 

< TT 

|A,- - A| 

k 

< TT 

lAj + Afc+i 

11 

1=1 J+*0 


- 11 

1=1 J++ 

1 Aj 1 “ 1A jQ 1 

- 11 

1=1 J++ 

o 

E 

1 


(5.6) 


Therefore, for z = 1, 2 ..., A; and |A| < |Afc+il, making use of Taylor series expansions, we get 


i^f^(A)i< n 


|Aj| + |Afc+i| 


|A,-|-|A 


^0 I 


n 


lAjr'l + |Afc+l| 


n 


|Aj| + |Afc+i| 


j^i |Aj| |Aio| j=iQ+i l'^*ol |Aj| 


*0^1 + 0 (e-(^-i+i)a) o (e-(fc-J+i)“) + 1 

11 1 _ 0 (g-(jo-f)“) 11 


i=i 

k 


j=io+l 


o (e(.?-*o)«) - 1 


n 


1 + 0 -l-jL 1 - O(e-ho-j>) O(e(.?'-*o)") - 1 


n 


n 


1 


(1 + O (e-(^-*o+b“)) J-J- 1 _ 0(e-(*o-i)“) 

n - " - 

i=*o+i '' U O (eO-*o)“) 

i=*o+i 

k+i \ / *0 \ ! fc—lo+i 

1 + E C> (e-('=-J+i)") 1 + E C>(e“^") 1 + E C>(e"^“) 


i=i 


i=i 


1=1 


(1 + 0 (e-(*:-++i)«)) 


(5.7) 


n O(e(i-*o)“) 

l=*o+l 


by absorbing those higher order terms into 0(.). Note that in the above numerator we have 
fc+i 


1 + 0(e-(^-^+i)“) = 1 + C> I e-(fc-l+i)« 1=1 + 0 

1=1 


( k+i 

S' 


1 — e 


— (1 — e 

—rv 


_ p-{k+l)a' 
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and 


*0 


*0 


l + ^0(e 

i=i 


-JO:\ _ 


1 + O ^ e-^“ =1 + 0 


vi=i 


1 — e 


— (1 — e 

—rv 


-10«' 


fc—iQ + l / fc—io + 1 

1 + ^ 0(e-^'“) = 1 + 0 e-^" 1=1 + 0 

i=i \ j=i 


1 — e 


3+(l 


_ p-(fc-*o+l)«' 


It is easy to check that for any 1 < io < k the product of the above three terms is no more than 

2 \ 


1 + 0 


3e- 


1 — e“ 


+ 0 


1 - e-“ 
k 


= 1 + 0(e- 


By definition, the factor 0{e^^ in the denominator of (5.7), which is 

j=io+l J=io+l ^ 

exactly one when ig = k, and it is bigger than one when io < k; the other factor 1 + 0 
is between 1 + 0 (e“^“) and 1 + 0 (e“"). Therefore, for any k and |A| < |Afc+i|, we have 


4'=)(A)| = l + 0(e-“), 


|l1 ^')(A)| = . rnax |Lr(A)| = l + 0(e-“). 

^ 1=1,2,...,k 


-{k), 


From this estimate and (5.5) it follows that 


\Bk 2 B^^\\^ < Vk{n - k) (1 + 0(e-“)) . 


(5.8) 

(5.9) 

(5.10) 


As a result, for /c = 1, 2,... , n — 1, from (5.4) we have 


|A| 


F < 


|Afc+i 


max 


-j=k+l I + 


vn\ 


|Afc| min^^Jn^6| 


yjk{n-k) (l + 0(e-“)) . □ 


Remark 5.1 Trivially, we have 


sin0(Vfc,V|)|| < 1. 


But in our context it is impossible to have || sin0(Vfc, V|)|| = 1 since is not a zero matrix. We 
have seen from the proof that the factor + , in it is intrinsic and unavoidable in 

^ |Afc| I'uj 6| 

(5.2). But the factor n{n — k) in (5.2) is an overestimate and can certainly be reduced. The 
reason is that (5.10) is an overestimate since \L\ TAj)| for i not near to k is considerably smaller 
than |Lj^^(Aj)|, j = A; + 1,..., n but we replace all them by the maximum \L\^\Xj)\ = 1 + 0(e“"). 
In fact, our derivation before (5.8) and (5.9) when replacing io by i clearly illustrates that the 
smaller i is, the smaller \L\^\Xj)\ than \L^i^\Xj)\, j = k + 1, ■ ■ ■ ,n. 

Recall the discrete Picard condition (1.4). Then the coefficients 


max; 


Cfc = 


Lk+i\vjb\ _ + vje\) ^ \Xk+i\^+^ + \vl^^e\ 


mmY\vJb\ mmYi\vJb + vJe\) + \vle\ 


(5.11) 


We see that, the larger /3 is, the smaller ~ , which is a constant for k < ko, and thus 

the better V| captures Vk- For k > ko, since all the \v'^b\ ^ \Vk^\ are roughly the same, we have 
Cfc ~ 1, meaning that may not capture Vk so well after iteration ko- 
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Remark 5.2 The theorem can be extended to moderately ill-posed problems with the singular 
values (Tj = a > 1 considerably and k not big, where the factor 1 -|- 0(e“") in (5.2) is 

replaced by a bigger 0(1). Let us look into why it is so. Recall that, by definition, |l|^^(A)| > 
|L^^^(A)| for |A| < |Afc+i|. Using a similar proof to that of Theorem 5.1 and the first order Taylor 
expansion, we can roughly estimate |L^^^(A)| as follows: 



k-l 


4‘’o)i < n 

i=i 


|Aj| -l- |Afc+i 
|A,|-|Afc| 


*-‘ i + o((A)‘‘) 

l-o((|)‘>) 


fe-1 


E 


(l + o 





0 ( 1 ). 


This estimate is not as accurate as that for severely ill-posed problems. More important is that it 
depends on k and increases slowly as k increases. The above estimate can be improved when A is 
symmetric definite: 




\L 


(")(A)i=n 


k-l 


|Aj — Afc+i| 




< 


|A, 


k-l 

0 |Ai - Afcl 


= rr_^_ 


smaller than the previous one. The two estimates mean that may capture better for A 
symmetric definite than for A symmetric indefinite where there are both positive and negative ones 
among the first A: -|- 1 large eigenvalues. 

Remark 5.3 A combination of (5.1) and (5.2) and the above analysis indicate that V| captures 
Vfc better for severely ill-posed problems than for moderately ill-posed problems. There are two 
reasons for this. The first is that the factors |Afc_|_i/Afc| are basically fixed constants for severely ill- 
posed problems as k increases, and they are smaller than the counterparts for moderately ill-posed 
problems unless the degree a of its ill-posedness is far bigger than one and k small. The second 
is that the factor 1 -I- 0{e~°‘) is smaller for severely ill-posed problems than the factor 0(1) for 
moderately ill-posed problems. 

Remark 5.4 The situation is fundamentally different for mildly ill-posed problems: Firstly, 
we always have |Lj^^(A)| > 1 substantially for |A| < |Afc+i|, a < 1 and any k. Secondly, Ck defined 
by (5.11) is closer to one than that for moderately ill-posed problems for k = 1,2,... ,kQ. Thirdly, 
for the same noise level ||e|| and j3, from the discrete Picard condition (1.4) and the definition of kQ 
we see that ko is bigger for a mildly ill-posed problem than that for a moderately ill-posed problem. 
All of them show that captures Vk considerably better for severely and moderately ill-posed 
problems than for mildly ill-posed problems. In other words, our results imply that contains 
substantial information on the other n — k eigenvectors for mildly ill-posed problems, causing that 
a small harmonic Ritz value generally appears for some k < ko, especially when /cq is not small. 
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Equivalently, the projected matrix generated by MR-II generally has a small singular value for 
some k < kQ, such that the solution is deteriorated, as deduced from (2.5). As a result, we are 
certain that MR-II has better regularizing effects for severely and moderately ill-posed problems 
than for mildly ill-posed problems. Most importantly, by this property, since MR-II has at most 
the full regularization for severely and moderately ill-posed problems, we deduce and are thus sure 
that MR-II generally has only the partial regularization for mildly ill-posed problems. 

We mention that, in comparison with the results, i.e., Theorem 2.1, in [13] on LSQR, we find 
that ICk{A,Ab) is as comparably effective as ICk{A'^A,A'^b), on which LSQR works, for capturing 
the ^-dimensional dominant eigenspace. 

Let us get more insight into the regularization of MR-II. Recall (2.5), where 

xW = {Qk+iTkQl)^b = QkTlQl+,b. 


Define 

7k = \\A-Qk+iTkQl\\, (5.12) 

which measures the quality or accuracy of the rank k approximation Qk+iThQ^ fo A. This quantity 
is central and fundamental to understand the regularizing effects of MR-II and measures how the 
iterates x^^^ by MR-II behave like the TSVD regularized solution Xk = Particularly, note 

that the best rank ko approximation satisfies ||A — Ak^W = (yk^+i- Then if 7 ^^ cr^o+i for 
(Tko+i reasonably small, Qko+i^koQ"kQ ^ near best rank fco approximation to A with approximate 
accuracy and has no small nonzero singular value. In this case, the regularized solution 

is close to the best TSVD regularized solution x^q, and MR-II has the full regularization. Otherwise, 
if 7 fco > ^ko+i considerably, then Qk^+iTk^Qj^ deviates from the best rank k approximation Ak^ 
considerably and x^^A jg ^ot close to Xkg, meaning that MR-II has only the partial regularization. 
Based on Theorem 5.1 and Remark 5.2, we can derive the following estimates for 7 ^. 

Theorem 5.2. Assume that (1.1) is severely or moderately ill posed. Then 

|Afc+i| < 7 fc < |Afc+i| -h |Ai|||sin0(Vfc,V|)||. (5.13) 

Proof. Note that Qk+iTkQ"k is of rank k. The lower bound in (5.13) is trivial since the best k 
approximation Ak to A satisfies ||A —A^jj = ak+i = |Afc+i|. We next prove the upper bound. From 
(2.3), we obtain 

||A - Qk+iTkQl\\ = ||A - AQkQlW = \\MI - QkQl)\\ ■ (5A4) 

From Theorem 5.1, it is known that = ICk{A,Ab) = span{Qk}. Let I 4 = {vi,V 2 ,... ,Vk) and 
Ak = diag(Ai, A 2 ,..., Xk)- Then by the definition of || sin 0(Vfc, V^)|| we obtain 

||A - AQkQlW = IK^ - VkAkV^ + VkAkV^){I - QkQl)\\ 

< ||(A - VkAkV^){I - QkQl)\\ + \\VkAkV^{I - QkQl)\\ 

< |Afc+i| + IjAfcll ||V)f(I — QkQ'k)\\ 

= |Afc+i| + |Ai|||sin0(Vfc,V|)||. □ 

Our later numerical experiments will indicate that 7 ^ ~ erk+i = |Afc+i| for severely and 
moderately ill-posed problems, illustrating that Qk+iTkQj is a near best rank k approximation 
to A with the approximate accuracy cTk+i. Particularly, since 7 ^^ « ^ko+i^ fh® MR-II iterate 
a;(^o) = is close to the TSVD solution Xkg provided that (Jko+i- Furthermore, we will 
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find that the error — xtmeW of MR-II iterate is as small as the error \\xkQ — xtmeW of the 

best TSVD solution This indicates that MR-II has the full regularization. Experimentally, 
for severely and moderately ill-posed problems, the observations 7 ^ (Jk+i appear to be general 
and thus should have strong theoretical supports. Our upper bound in (5.13) appears to be a 
considerable overestimate. 

Recall that a* and jSi, i = 1,2,... ,k denote the diagonals and subdiagonals of dehned by 
(2.3), respectively. We next establish some interesting and intimate relationships between them 
and 7 fc, showing how fast and /3k decay. 

Theorem 5.3. For k = 1,2,... ,n — 2 we have 

/3fc+i < 7fc; (5-15) 

|afc+2| < 7fc- (5-16) 


Proof. Since the Lanczos process can be run to completion, we have 


Qn^Qn — Trij 


where Qn G is orthogonal, and 


T = 

r) — 


/ ai /3i 
/3l 02 h 


V 


/3n-l 

/3n—l On j 


is symmetric tridiagonal. Thus, from (2.3) we have 

Ik = 11^ — Qk+lTkQk\\ = ||Qn “ Qk+lTkQk)Qr 


where 


f„- ( 0 )r7 7 0 ) 


/ I3k+l Ok+2 /3fc+2 

h+2 afc+3 h+i 


Gk — 


= IlCfcll, 

\ 


G 


[n—k—l) X [n—k) 


/3n-l 

/3n—1 On j 

from which and > 0 , i = 1 , 2 ,... , n — 1 it follows that 


(5.17) 


h+i — IIG'fceill < ||Gfc|| — 7fe 

and 

l“fc-|-2| < “1+2 + /^fc+2 = < \\Gk\\ = Ik 

for A; = 1, 2,..., n — 2. Therefore, (5.15) and (5.16) hold. □ 

This theorem indicates that |afc-i- 2 | and I3k+i decay at least as fast as 7 ^. Moreover, based on 
the experimental observations that 7 ^, Ok+i for severely and moderately ill-posed problems, the 
theorem illustrates that |afc-i- 2 | and /3fc-i-i decay as fast as cr^+i, A; = 1, 2,... , n — 2, for these two 
kinds of problems. 
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6 Numerical experiments 


In this section, we report numerical experiments to illustrate the regularizing effects of MINRES 
and MR-II and make a number of comparisons. We justify our theory; (i) MINRES has only the 
partial regularization, independent of the degree of ill-posedness, and a hybrid MINRES is generally 
needed; (ii) the feth MINRES iterate is always more accurate than the {k — l)th MR-II iterate 
until the semi-convergence of MINRES; (iii) MR-II has only the partial regularization for 
mildly ill-posed problems, and a hybrid MR-II is needed. In the meantime, experimentally, we 
demonstrate a remarkable and attractive feature, stronger than our theory predicts: MR-II has 
the full regularization for severely and moderately ill-posed problems and its iterates at semi¬ 
convergence is as accurate as the best TSVD solutions for these two kinds of problems. We will use 
the function Icurve in [9] to depict the L-curves. In order to simulate exact arithmetic, the Lanczos 
process with reorthogonalization is used in MINRES and MR-II. 

Table 1 lists test problems and their degree of ill-posedness, all of which are symmetric and arise 
from the discretization of the first kind Eredholm integral equations; see Hansen’s regularization 
toolbox [9] for details. Eor each problem except the 2D image deblurring problem ‘blur’, we use 
the corresponding code in [9] to generate a 1024 x 1024 matrix H, the true solution xtme and 
noise-free right-hand h. In order to simulate the noisy data, we generate the white noise vector e 
whose entries are normally distributed with mean zero, such that the relative noise level e = Ijsll = 

10“^, 10“^, 10“'^, respectively. To simulate exact arithmetic, the full reorthogonalization is used 
during the Lanczos process. We remind that, as far as ill-posed problem (1.1) is concerned, our 
primary goal consists in justifying the regularizing effects of iterative solvers, which are unaffected 
by sizes of ill-posed problems and only depends on the degree of ill-posedness. Therefore, for 
this purpose, as extensively done in the literature (see, e.g., [8, 10] and the references therein), 
it is enough to test not very large problems. Indeed, for n large, say, 1,0000, we have observed 
completely the same behavior as that for n not large, e.g., n = 1024 used in this paper except for 
the problem ‘blur’ with n = 65,536. A reason for using n not large is because such choice makes 
it practical to fully justify the regularization effects of LSQR by comparing it with the TSVD 
method, which suits only for small and/or medium sized problems for computational efficiency. All 
the computations are carried out using Matlab 7.8 with the machine precision Cmach = 2.22 x 10“^® 
under the Microsoft Windows 7 64-bit system. 


Table 1: The description of test problems. 


Problem 

Description 

Ill-posedness 

shaw 

One-dimensional image restoration model 

severe 

foxgood 

Severely ill-posed test problem 

severe 

gravity 

One-dimensional gravity surveying problem 

severe 

phillips 

phillips’ ’’famous” test problem 

moderate 

deriv2 

Computation of second derivative 

mild 

blur 

2D Image deblurring test problem 

mild/moderate 


6.1 A comparison of the regularizing effects of MR-II and MINRES 

We now compare MINRES and MR-II and justify our theory; (i) the MR-II iterate is always more 
accurate than the MINRES iterate at their respective semi-convergence, meaning that MINRES 
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(c) (d) 


Figure 1: (a)-(b); The relative errors — a:trMe||/|| 2 :trMe|| by MINRES and MR-II; (c)-(d): Plots 
of the singular values (circles for MINRES, stars for MR-II) of the projected matrices and the ones 
(solid lines) of A for shaw (left) and foxgood (right). 


cannot obtain best possible regularized solutions and has only the partial regularization, indepen¬ 
dent of the degree of ill-posedness; (ii) the MINRES iterates are always more accurate that the 
MR-II iterates until the semi-convergence of MINRES. 

In this subsection, we only report the results for the noise level e = 10“^. Results for the other 
two £ are analogous and thus omitted. 

Figures 1 and 2 display numerous curves for severely and moderately ill-posed problems. Clearly, 
all the MR-II ierates are always more accurate than the MINRES iterates at their respective semi¬ 
convergence. This indicates that MINRES has only the partial regularization. As elaborated 
previously, this is because that a small singular value of the projected matrix T). appears before a 
regularized solution becomes best, causing that its error does not reach the same error level as that 
obtained by MR-II. For instance, we see from Figure 1 (a) and (c) that all the singular values of 
Tk in MR-II are excellent approximations to the k large singular values of A in natural order for 
k < 9. We see that the semi-convergence of MR-II occurs at iteration k = 7. By the comments in 
the end of Section 2 and the explanations after (5.12), this clearly justifies the full regularization 
of MR-II, and the best possible regularized solution by MR-II includes seven dominant spectral or 
SVD components. On the other hand, it is clearly seen from Figure 1 (c) that the smallest singular 
value of T 5 in MINRES is smaller than erg = Cfco+i, making the relative error starts to increase 
dramatically at iteration 5 and MINRES have only the partial regularization. 

Similar phenomena are observed for foxgood, and MR-II has the full regularization with k^ = 3, 
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(c) 


(d) 


Figure 2: (a)-(b): The relative errors — iCtTOell/H^^fruell by MINRES and MR-II; (c)-(d): Plots 
of the singular values (circles for MINRES, stars for MR-II) of the projected matrices and the ones 
(solid lines) of A for gravity (left) and phillips (right). 


as indicated by Figure 1 (b) and (d), where the smallest singular value of T 3 lies between <74 and 
<75 and the best iterate by MINRES at semi-convergence is considerably less accurate than 
the best iterate by MR-II at semi-convergence, meaning that MINRES has only the partial 
regularization. We have analogous findings for gravity and phillips, as shown by Eigure 2 (b) and 
(d), which again demonstrate that MR-II has the full regularization but MINRES has only the 
partial regularization. 

As for the mildly ill-posed problem deriv 2 , we also see from Eigure 3 (a) that the relative error 
obtained by MR-II clearly reaches the lower minimum level than that by MINRES, indicating that 
MR-II has better regularizing effects than MINRES. 

The above experiments have illustrated that MR-II always obtains more accurate regularized 
solutions than MINRES does for the test severely, moderately and mildly problems. This justifies 
our theory that MINRES only has the partial regularization, independent of the degree of ill- 
posedness. Therefore, one must use a hybrid MINRES with some regularization method applied 
to the projected problems in order to remove the effects of small singular values of and improve 
the accuracy of regularized solutions until a best regularized solution is found. 

It is clear from Eigures 1-2 and Figure 3 (a) that, for each test problem, the MINRES iterates 
^(fc) more accurate than the corresponding MR-II iterates until the semi-convergence 

of MINRES. Afterwards, the regularized solutions x^^^ are deteriorated more and more seriously. 
This confirms our theory in Section 4, i.e., assertion (ii) in the beginning of this subsection. 
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L-curve 




Figure 3: (a): The relative errors — Xiruell/II^Jiraell by the pure MINRES and MR-II as well 
as the hybrid MINRES and MR-II; (b): The L-curves of MINRES and MR-II for deriv2. 

6.2 The regularizing effects of MR-II, MINRES and their hybrid variants 

We hrst test MR-II, MINRES and their hybrid variants for the mildly ill-posed problem deriv2, 
and justify our theory that MR-II has only the partial regularization and one must use its hybrid 
variant to compute a best possible regularized solution. 

For deriv2, Figure 3 (a) shows that the relative errors of regularized solutions obtained by the 
hybrid MINRES and MR-II with the TSVD regularization method applied to the projected prob¬ 
lems reach a considerably smaller minimum level than those by MINRES and MR-II themselves. 
For this problem, before MINRES or MR-II captures all the dominant spectral components needed, 
a small singular value of Tk or appears and starts to deteriorate the regularized solutions. In 
contrast, their hybrid variants expand Krylov subspaces until enough dominant spectral compo¬ 
nents are captured and the TSVD regularization method effectively dampens the SVD components 
corresponding to small singular values of the projected matrices by MINRES and by MR-II. 
For example, we see from Figure 3 (a) that the semi-convergence of MR-II occurs at iteration k = 3, 
which is also observed by the corner of the L-curve depicted by Figure 3 (b). However, as shown 
by Figure 3 (a), such regularization of MR-II is not enough, and the hybrid MR-II uses a larger six 
dimensional Krylov subspace JCq{A, Ab) to improve the solutions and get a best possible regularized 
solution, whose residual norm is smaller than that obtained by the pure MR-II. After k = Q, the 
regularized solutions almost stabilize with the minimum error as k increases. We observe similar 
phenomena for MINRES and its hybrid variant, where we find that the relative error by the hybrid 
MINRES reaches the same minimum level as that by the hybrid MR-II. 

Next we test MR-II, MINRES and their hybrid variants for severely and moderately ill-posed 
problems. We attempt to get more insight into the regularizing effects of MR-II. As a matter 
of fact, we have already justified the full regularization of MR-II for the four test problems in 
Section 6.1. In what follows, we will give more details and justifications on the full regularization 
of MR-II. We show that (i) the relative error obtained by the hybrid MINRES reaches the same 
minimum level as that by the hybrid MR-II; (ii) MR-II has the full regularization effects: at semi¬ 
convergence, the regularized solution by the pure MR-II is as accurate as that by the hybrid MR-II 
with the TSVD regularization used within projected problems; (hi) MR-II generates near best rank 
k approximations Qk+iTkQ^ to i-®-) the relation 7 ^ < 7^+1 = |Afc+i| holds with different noise 

levels. Keep in mind (1.5) and (2.5). This means that Qk+iTkQ]^ generated by MR-II plays the 
same role as Ak, the best rank k approximation to A, so that MR-II has the full regularization. 
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(a) (b) 




(c) 


(d) 


Figure 4: The relative errors —xtrue\\/\\xtrue\\ by MR-II, and hybrid MR-II and MINRES with 
additional TSVD regularization for shaw, foxgood, gravity, phillips (from top left to bottom right). 


For MR-II and the hybrid MR-II, we observe from Figure 4 that MR-II reaches the same error 
level as the hybrid MR-II, and the TSVD regularization applied to projected problems simply 
makes the regularized solutions with the minimum error almost stabilize and does not improve the 
regularized solution by MR-II at semi-convergence. This justifies the full regularization of MR-II. 

Compared with Figures 1-2, we find from Figure 4 that the hybrid MINRES improves on 
MINRES substantially and the relative errors of iterates by the hybrid MINRES reach the same 
minimum level as MR-II and the hybrid MR-II. These phenomena again justify our assertion in 
Section 4 that the hybrid MINRES is necessary, independent of the degree of ill-posedness, and the 
hybrid MINRES is as effective as the hybrid MR-II. 

Figure 5 and Figure 7 display the curves of sequences 7 ^ with the noise levels e = 10“^, 10“^, 10“"^ 
respectively, for the four severely and moderately problems. We see that 7 ^ (Tk+i = |Aa:+i|, al¬ 
most independent of noise level e. We point out that, due to the round-offs in finite precision 
arithmetic, they level off at the level of Cmach when k = 20 for shaw, k = 37 for foxgood and k = 50 
for gravity. The results indicate that the Qk+iTkQ'^ are near best rank k approximations to A 
with the approximate accuracy ak+i so that Tk does not become ill-conditioned before k < k^. 
As a result, the regularized solutions become increasingly better approximations to xtme until 
iteration /cq, and they are deteriorated after that iteration. At iteration fco, captures the fcg 
dominant spectral or equivalent SVD components of A and is a best possible regularized solution, 
i.e., MR-II has the full regularization for the severely ill-posed problems tested. 

Figure 6 and Figure 8 plot the relative errors — xtrjie|| /HiCtruell with different noise levels 
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(a) 


(b) 




(c) 


(d) 


Figure 5: (a)-(b): Plots of decaying behavior of the sequences 7 ^ and |Afc_|_i| for shaw with e = 10“^ 
(left) and e = 10“^ (right) by MR-II; (c)-(d): Plots of decaying behavior of the sequences 7 ^ and 
|Afc+i| for foxgood with e = 10“^ (left) and e = 10“'^ (right) by MR-II. 


for these four severely and moderately ill-posed problems. For smaller noise levels, MR-II gets more 
accurate best regularized solutions at cost of more iterations. This is expected since, from (1.4) 
and |A^+fj =1 '^fcQ+i^ l~l '^fco+ 1 ^ l> a bigger is needed for a smaller noise level ||e||. Moreover, 
MR-II needs more iterations to achieve semi-convergence for moderately ill-posed problems with 
the same noise level, since (jj does not decay as fast as that for a severely ill-posed problem. 

Figures 9 display the decreasing curves of quantities |afc-i-i|) h and cjfc, fc = 2,... ,n — 1. From 
Figure 9 (a), we see that, for the severely ill-posed problem shaw, |Q!fc+i| and decrease as fast as 
and the three quantities level off at the level of emach for k no more than 20 , and after that these 
quantities are purely round-offs and not reliable any more. Similar phenomena are also observed 
for the other two severely ill-posed problems foxgood and gravity, as indicated by Figure 9 (b) and 
(c). Figure 9 (d) illustrates that /3fc decreases as fast as Cfc but |afc+i| decays as fast as ak in the 
first iterations and then considerably faster than cr^ as k increases in the later stage for moderately 
ill-posed problem phillips. 

Finally, we report some comparison results on LSQR, the hybrid LSQR and MR-II, the hybrid 
MR-II. As already proved in [13], a hybrid LSQR should be used to compute best possible regular¬ 
ized solutions for mildly ill-posed problems. It has also been experimentally justified in [13] that 
LSQR has the full regularization for severely and moderately ill-posed problems. We have tested 
LSQR and the hybrid LSQR, and compared their effectiveness and efficiency with MR-II and the 
hybrid MR-II. We have found that, for each of the severely and moderately ill-posed problems in 
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Figure 6; The relative error — xt,.ue|| /Ha^truell with respect to e = 10 10 10 ^ for shaw 

(left) and foxgood (right) by MR-II. 




(a) (b) 




Figure 7; (a)-(b): Plots of decaying behavior of the sequences jk and |Afc+i| for gravity with 
e = 10“^ (left) and e = 10“^ (right) by MR-II; (c)-(d): Plots of decaying behavior of the sequences 
7 fc and |Afc+i| for phillips with e = 10“^ (left) and e = 10“^ (right) by MR-II. 
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(a) (b) 

Figure 8: The relative errors — XirMe||/llaJiruell with respect to e = 10“^, 10“^, 10“^ for 

gravity (left) and phillips (right) by MR-II. 


Table 1 and with the same noise level, both the pure MR-II and LSQR obtain the best regularized 
solutions with almost the same accuracy using almost the same iterations. For deriv2, the hybrid 
MR-II and LSQR compute the best possible regularized solutions using almost the same iterations. 
These results tell us two things: (i) As an iterative regularization method, MR-II is as effective as 
LSQR for an ill-posed problem; (ii) MR-II is twice as efficient as LSQR. 

6.3 A 2D image restoration problem 

The problem blur is a 2D image deblurring problem and more complex than the other five ID 
problems in Table 1. It arises in connection with the degradation of digital images by atmospheric 
turbulence blur. We use the code blur(n, band, sigma) in [9] to generate an x v? matrix A, the 
true solution xtrue and noise-free right-hand b. The vector xtme is a columnwise stacked version of 
a simple test image, while b = Axtme holds for a columnwise stacked version of the blurred image. 
The blurring matrix A is block Toeplitz with Toeplitz blocks, which has two parameters band and 
sigma; the former specifies the half-bandwidth of the Toeplitz blocks, and the latter controls the 
shape of the Gaussian point spread function and thus the amount of smoothing. We generate a 
blurred and noisy image 6 = 6 -|- e by adding a while noise vector e. The goal is to restore the true 
image xtrue from h. 

We take n = 256 and the relative noise level e = 5 x 10“^, giving rise to A with order = 
65,536. It is known that the larger the sigma, the less ill-posed the problem. Purely for an 
experimental purpose, we computed all the singular values of a few A with < 10,000 using 
the matlab function svd. Since the degree of ill-posedness is the same for different large re^, we 
have deduced from the computed singular values for these matrices A that band = 3, sigma = 0.7 
(the default setting) generates mildly ill-posed problems, while band = 7, sigma = 2 gives rise to 
moderately ill-posed problems. We next test MINRES, MR-II and their hybrid variants for these 
two problems, and verify the regularizing effects similar to the previous mildly and moderately 
ill-posed problems: deriv2 and phillips. 

Figure 10 (a) shows that MINRES and MR-II have the partial regularization for the mildly 
ill-posed problem blur. The semi-convergence of the two methods occurs at the very first iteration, 
then regularized solutions are progressively deteriorated, while the hybrid MINRES finds the best 
possible regularized solution at iteration k = 9 and the hybrid MR-II does so at A: = 8. Moreover, we 
see that the hybrid MINRES and MR-II reaches the same minimum error level. Figure 10 exhibits 
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Figure 9: (a)-(d): Plots of decaying behavior of the sequences |afc+i|, fik and at for shaw, foxgood, 
gravity, phillips (from top left to bottom right) with e = 10“^ by MR-II. 


the restoration performance, where the restored image is chosen by the regularized solution at the 
iteration where the hybrid MR-II first reaches the minimum error level. We observe from Figure 10 
(d) that the outline of original image is restored quite well by the restored image. 

From Figure 11 (a), we see that the semi-convergence of MR-II occurs at the first iteration and 
the regularized solution at this iteration is as accurate as those obtained by the hybrid MR-II and 
MINRES for the moderately ill-posed problem blur. Therefore, MR-II has the full regularization 
for this problem. In contrast, MINRES has only the partial regularization because its regularized 
solution at semi-convergence is much less accurate than that obtained by MR-II. In addition, we 
observe that the hybrid MINRES and the hybrid MR-II simply make the regularized solutions 
almost stabilize with the minimum error. Figure 11 (d) exhibits the restored image, which is a 
good approximation to the original image. 

7 Conclusions 

For large scale symmetric discrete linear ill-posed problems, MINRES and MR-II are natural al¬ 
ternatives to LSQR. Our theory and experiments have shown that MINRES has only the partial 
regularization and its hybrid variant is needed to find best possible regularized solutions, inde¬ 
pendent of the degree of ill-posedness. We have proved that MR-II has better regularizing effects 
for severely and moderately ill-posed problems than for mildly ill-posed problems, and it generally 
has only the partial regularization for mildly ill-posed problems. We have shown that although 
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Figure 10: (a): The relative errors — Xiruell/ll^^iruell by MINRES, hybrid MINRES, MR-II, 
and hybrid MR-II; (b): The original image; (c): The blurred and noisy image; (d): The restored 
image with band = 3, sigma = 0.7. 


MR-II is a better regularization method than MINRES, the kth. MINRES iterate is always more 
accurate than the {k — l)th MR-II iterate until the semi-convergence of MINRES. We have also es¬ 
tablished estimates for the entries generated by the Lanczos process working on 1C{A, Ab), showing 
how fast they decay. All these results have been confirmed numerically. Remarkably, stronger than 
our theory predicts, we have numerically demonstrated that MR-II has the full regularization for 
severely and moderately ill-posed problems and can compute best possible regularized solutions. 
As a comparison of MR-II and LSQR for a general symmetric ill-posed problem, our theory exper¬ 
iments have indicated that two methods have very similar regularizing effects but MR-II is twice 
as efficient as LSQR, so do their hybrid variants. Therefore, for a large scale symmetric problem 
(l.I), MR-II may be preferable to LSQR. 

Some problems need to be further considered. As we have seen, more appealing is a sharp 
estimate for ||Afc|| other than ||Afc||i?. The quantity || sin 0(Vfc, V|)|| needs a more subtle analysis 
and plays a central role in accurately estimating the accuracy 'jk of the rank k approximation 
generated by the Lanczos process working on /C(A, Ah). As we have elaborated, studying how near 
7 fc is to (Tfc+i is a central problem that completely understands the regularizing effects of MR-II. 
Our bounds in Theorems 5.1-5.2 are less sharp and need to be improved on. 
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Original image 



(c) (d) 

Figure 11: (a): The relative errors — xtriiell/Ha^traell with respect to MINRES, hybrid MINRES, 

MR-II, and hybrid MR-II; (b): The original image; (c): The blurred and noisy image; (d): The 
restored image with band = 7, sigma = 2. 
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